Camera-based orientation fix from portrait to landscape

ABSTRACT

A portable electronic device includes a camera and a display switchable between a portrait mode and a landscape mode in response to an inertial sensor. When it is determined that the display mode should be switched in response to movement of the device, a feature of a user of the device, such as the user&#39;s face, is imaged with the camera. If the imaged feature changes orientation with respect to the display by less than a threshold amount, then the switch in display mode is disabled and the display remains in its current mode. Other embodiments are also described and claimed.

BACKGROUND

1. Field

The invention relates generally to the field of displays for portablehandheld electronic devices and, more particularly, to a display that isswitched between portrait and landscape modes as a function of a sensedorientation of the user.

2. Background

Portable data and media display devices, such as mobile telephones,e-book readers, tablet computers and the like typically have rectangulardisplay screens with a “portrait” orientation. It is often desirable toview content in a “landscape” orientation. This can be accomplished byrotating the display screen by 90° and selecting a landscapepresentation (or landscape display mode). A manual selection may beprovided, but many devices now have inertial sensors to automaticallyselect a landscape presentation when rotation of the device is sensed.

In some situations, the user of a portable device may not want thedisplay mode to change when the device is rotated. For example, a usermay start reading email messages or playing a game while sittingupright, but may then decide to lie down while continuing to use thedevice. The user does not want the display mode to change in such asituation because the relative orientation of the user's eyes withrespect to the display screen has not changed. While a manual displaymode selection may be provided to override the inertial sensor, this isan inconvenient solution to the problem.

SUMMARY

An embodiment of the present invention is a mode of operation in aportable electronic device, for stabilizing the display of the devicewith respect to the user's face. The portable electronic device has acamera and a display screen for which the display mode is switchablebetween a portrait mode and a landscape mode in response to an inertialsensor. When it is determined that the display mode should be switchedin response to movement of the device, a feature of a user of thedevice, such as the user's face, is imaged with the camera. If theimaged feature changes orientation with respect to the device by lessthan a threshold amount, then the switch in display mode is disabled sothat the display mode remains in its current mode.

The above summary does not include an exhaustive list of all aspects ofthe present invention. It is contemplated that the invention includesall systems and methods that can be practiced from all suitablecombinations of the various aspects summarized above, as well as thosedisclosed in the Detailed Description below and particularly pointed outin the claims filed with the application. Such combinations haveparticular advantages not specifically recited in the above summary.

BRIEF DESCRIPTION OF THE DRAWINGS

The embodiments of the invention are illustrated by way of example andnot by way of limitation in the figures of the accompanying drawings inwhich like references indicate similar elements. It should be noted thatreferences to “an” or “one” embodiment of the invention in thisdisclosure are not necessarily to the same embodiment, and they mean atleast one.

FIGS. 1A-1C illustrate a device with portrait and landscape displaymodes selected in accordance with an embodiment of the presentinvention.

FIG. 2 is a functional flow diagram of a display mode selection processin accordance with an embodiment of the present invention.

FIG. 3 is a functional block diagram of an electronic device in whichembodiments of the present invention may be practiced.

DETAILED DESCRIPTION

Throughout the description, for the purposes of explanation, numerousspecific details are set forth in order to provide a thoroughunderstanding of the present invention. It will be apparent, however, toone skilled in the art that the present invention may be practicedwithout some of these specific details. In other instances, well-knownstructures and devices are shown in block diagram form to avoidobscuring the underlying principles of embodiments of the presentinvention.

Portable handheld electronic devices, such as the iPod™ and iPhone™multifunction devices by Apple Inc., have a display screen for viewingphotos, videos, documents, web content and other media. A device of thistype, denoted as 10, is shown in FIG. 1A. The display screen 12 istypically configured with a height greater than the width in the normalorientation of the device so that objects on the screen are said to bedisplayed in “portrait” display mode as shown in FIG. 1A. While thedevice 10 is being held in this orientation, the text and other contentis displayed horizontally which is consistent with the horizontalorientation of user 20's eyes. Device 10, as is the case with manysimilar devices, has a built-in accelerometer or similar inertial sensorto detect the movement and orientation of the device. With thiscapability and suitable display software, the device may be rotated 90°,and objects on the display screen will then be presented in “landscape”display mode as shown in FIG. 1B. Text and other content remainsdisplayed horizontally.

Referring again to FIG. 1A, device 10 includes a built-in front-facingdigital camera 14. The imaging lens 110 (see FIG. 3) of the camera 14 isaimed out of the front face of the device 10, in which the displayscreen 12 is located. As will be more fully explained below, camera 14is used to capture an image of the device user 20 while the user isfacing the display screen 12, in order to override the usual selectionof the portrait or landscape display mode in certain situations. As seenin FIG. 1B, the selection of the landscape mode upon rotation of thedevice 10 by 90° is appropriate for the orientation of user 20. In FIG.1C, however, the user is reclining and the user's eyes are now orientedalong a generally vertical line as opposed to the horizontal orientationin FIGS. 1A and 1B. Here, it is appropriate to maintain the display inthe portrait mode even though device 10 has been rotated 90° as comparedto FIG. 1A since the displayed text and other content will remainproperly oriented with respect to the user's eyes.

A process for implementing an embodiment of the present invention isshown in FIG. 2. The process can be performed by a suitably programmedprocessor in the device 10 (e.g., an applications processor). Theprocess begins with user selection of a mode of operation of theportable device that stabilizes the display with respect to the user'sface at 20. This may be accomplished with a menu selection, by launchinga special purpose application, by depressing a manual switch or by othermeans known to those skilled in the art. The process may beautomatically selected in certain applications. An image of the user'sface is captured with the front-facing camera 14 and is then analyzed at22 to establish a starting reference for the user's orientation. Thismay be done, for example, by determining the orientation of a linepassing through the user's eyes in the plane of the image, relative tothe orientation of the portable device (e.g., its display screen) atthat time. The latter may be considered to be the reference orientationof the device and can be determined using an inertial sensor (e.g., anaccelerometer, a multi-axis turn rate gyro) in the device, in accordancewith known techniques. Facial analysis algorithms suitable for thepurposes of this invention, including recognizing the user's eyes in theimage, are also known to those skilled in the art. For example, theiPhoto™ application by Apple Inc. uses facial detection to identifyfaces of people in photographs. For instance, the relative orientation(user's eyes relative to display screen) may be as in FIG. 1A and may bestored as the angle computed between a straight line crossing the user'seyes in the captured image, and the X-axis of the camera's image sensor.

The device's accelerometer or other integrated inertial sensor ismonitored at 24 to determine when the device 10 has been tiltedsufficiently to otherwise trigger a switch of the display mode fromportrait to landscape or vice versa. This may be an entirelyconventional operation. If a tilt is detected in such operation, animage of the user's face captured with the front-facing camera at thatpoint is again analyzed at 26 to determine if the user's orientation haslikewise changed. If the orientation of the device has tilted without acorresponding change in the user's orientation, then the display mode isswitched at 28. On the other hand, if the user's orientation has alsochanged such that the user's face remains in approximately the sameorientation relative to the display screen, then the current displaymode is maintained at 30. In other words, if the image feature of theuser's face changes orientation with respect to the device's orientationby less than a threshold amount (e.g., a predetermined angle such as45°), then the switching of the display mode is disabled. Thus, in theexample above, if the line crossing the user's eyes remains within 45°of the image sensor's X-axis, then the user's orientation has notchanged enough to allow the display mode to be switched. Multiple imagesof the user, which may be captured by the same camera 14 as the device10 is tilted from one orientation to another (e.g., horizontal tovertical or vice versa), may be analyzed to make the determination as tohow much the user's feature changes its orientation.

FIG. 3 depicts a block diagram of an exemplary portable handheldmultifunction device 10 in which an embodiment of the invention may beimplemented. The device 10 has a processor 102 that executesinstructions to carry out operations associated with the device 10. Theinstructions may be retrieved from memory 104 and, when executed,control the reception and manipulation of input and output data betweenvarious components of device 10. Although not shown, the memory 104 maystore an operating system program that is executed by the processor 102,and one or more application programs are said to run on top of theoperating system to perform specific functions. The display screen 106displays a graphical user interface (GUI) that allows a user of thedevice 10 to interact with various application programs running in thedevice 10. The GUI displays icons or graphical images that representapplication programs, files, and their associated commands on the screen106. These may include windows, fields, dialog boxes, menus, buttons,cursors, scrollbars, etc. The user can select from these graphicalimages or objects to initiate the functions associated therewith.

In one embodiment, the screen 106 is a touch screen that also acts as aninput device to transfer data from the outside world into the device 10.This input is received via, for example, the user's finger touching thesurface of the screen 106, but it may also be received via physicalbuttons on the device 10.

Camera functionality of the device 10 (or camera 14, see FIG. 1) may beenabled by the following components. One or more solid state imagesensors 108 are built into the device 10 and each may be located at afocal plane of an optical system that includes a respective lens 110. Anoptical image of a scene within the camera's field of view is formed onthe image sensor 108, and the sensor 108 responds by capturing the scenein the form of a digital image or picture consisting of pixels that willthen be stored in memory 104. The image sensor 108 may include a solidstate image sensor chip with several options available for controllinghow an image is captured.

The device 10 may operate not just in a digital camera mode, but also inmobile voice and/or data telecommunications modes. These are enabled bythe following components. An integrated antenna 112 that is driven andsensed by RF circuitry 114 is used to transmit and receive cellularnetwork communication signals from a nearby base station (not shown). Amobile phone application executed by the processor 102 presents mobiletelephony options on the screen 106 for the user, such as a virtualtelephone keypad with call and end buttons. The mobile phone applicationalso controls at a high level the two-way conversation in a typicalmobile telephone call, by directing a speech signal from the built-inmicrophone 116 to the RF circuitry, while at the same time directing aspeech signal from the other side of the conversation through thereceiver or ear speaker 118. The mobile phone application also respondsto the user's selection of the receiver volume, by detecting actuationof a physical volume control 120. Although not shown, the processor 102may include a cellular base band processor that is responsible for muchof the digital audio signal processing functions and cellular networkprotocol signaling associated with a cellular phone call, includingencoding and decoding the voice signals of the participants to theconversation. Likewise, processor 102 may include a data communicationsprocessor that is responsible for encoding and decoding, and/orpacketization and depacketization of data, for transmission andreception over one or more wireless data networks.

The device 10 may be placed in any of its available modes of operation,such as a mobile telephone mode, a digital camera mode or a web browsingmode, in response to, for example, the user actuating a physical menubutton 122 and then selecting an appropriate icon on the display deviceof the screen 106. Any of various application programs stored in memory104 may be similarly selected by the user.

The device 10 includes an inertial sensor 124 such as an accelerometer,which may be a three-dimensional accelerometer that detects accelerationin three axial directions of the device 10, i.e. the up-down direction(Y-axis), the left-right direction (X-axis), and the front-reardirection (the Z-axis). The accelerometer allows the inclinations andmovements of the device 10 in the X-axis, Y-axis and Z-axis directionsto be determined. In an alternative embodiment, a rate gyro may beemployed that detects rotation of the device 10 about the Z-axis.Acceleration or turn rate data detected by the sensor 124 is transmittedto the processor 102 for processing to enable determination of theposition, orientation, and/or movement of the device 10. Thisinformation may then be processed to select between portrait andlandscape display modes when the orientation of device 10 reachespredetermined limits.

It will be apparent from this description that aspects of the presentinvention may be embodied, at least in part, in software, hardware,firmware, or in combination thereof. That is, the techniques may becarried out in a computer system or other data processing system inresponse to its processor, such as a microprocessor, executing sequencesof instructions contained in a memory, such as ROM, volatile RAM,non-volatile memory, cache, or a remote storage device. In variousembodiments, hardwired circuitry may be used in combination withsoftware instructions to implement the present invention. Thus, thetechniques are not limited to any specific combination of hardwarecircuitry and software or to any particular source for the instructionsexecuted by the data processing system. In addition, throughout thisdescription, various functions and operations are described as beingperformed by or caused by software code to simplify description.However, those skilled in the art will recognize that what is meant bysuch expressions is that the functions result from execution of code bya processor, such as a microprocessor.

A machine-readable medium can be used to store software and data whichwhen executed by the data processing system causes the system to performvarious methods of the present invention. This executable software anddata may be stored in various places including for example ROM, volatileRAM, non-volatile memory, and/or cache. Portions of this software and/ordata may be stored in any one of these storage devices.

The invention also relates to apparatus for performing the operationsherein. This apparatus may be specially constructed for the requiredpurposes, or it may comprise a general purpose computer selectivelyactivated or reconfigured by a computer program stored in the computer.Such a computer program may be stored or transmitted in amachine-readable medium. A machine-readable medium includes anymechanism that provides (i.e., stores and/or transmits) information in aform accessible by a machine (e.g., a computer, network device, personaldigital assistant, manufacturing tool, any device with a set of one ormore processors, etc.). For example, a machine readable medium includesrecordable/non-recordable media such as, but not limited to, amachine-readable storage medium (e.g., any type of disk including floppydisks, optical disks, CD-ROMs, and magnetic-optical disks, read-onlymemories (ROMs), random access memories (RAMs), EPROMs, EEPROMs, flashmemory, magnetic or optical cards, or any type of media suitable forstoring electronic instructions), or a machine-readable transmissionmedium such as, but not limited to, any type of electrical, optical,acoustical or other form of propagated signals (e.g., carrier waves,infrared signals, digital signals, etc.).

Additionally, it will be understood that the various embodimentsdescribed herein may be implemented with a variety of data processingsystems. For example, such data processing systems may be a cellulartelephone or a personal digital assistant (PDA) or an entertainmentsystem or a media player (e.g., an iPod) or a consumer electronicdevice, etc., each of which can be used to implement one or more of theembodiments of the invention.

Throughout the foregoing specification, references to “one embodiment,”“an embodiment,” “an example embodiment,” etc., indicate that theembodiment described may include a particular feature, structure, orcharacteristic, but every embodiment may not necessarily include theparticular feature, structure, or characteristic. Moreover, such phrasesare not necessarily referring to the same embodiment. When a particularfeature, structure, or characteristic is described in connection with anembodiment, it is submitted that it is within the knowledge of oneskilled in the art to bring about such a feature, structure, orcharacteristic in connection with other embodiments whether or notexplicitly described. Various changes may be made in the structure andembodiments shown herein without departing from the principles of theinvention. Further, features of the embodiments shown in various figuresmay be employed in combination with embodiments shown in other figures.

In the description as set forth above and claims, the terms “coupled”and “connected,” along with their derivatives, may be used. It should beunderstood that these terms are not intended to be synonymous with eachother. Rather, in particular embodiments, “connected” is used toindicate that two or more elements are in direct physical or electricalcontact with each other. “Coupled” may mean that two or more elementsare in direct physical or electrical contact. However, “coupled” mayalso mean that two or more elements are not in direct contact with eachother, but yet still co-operate or interact with each other.

Some portions of the detailed description as set forth above arepresented in terms of algorithms and symbolic representations ofoperations on data bits within a computer memory. These algorithmicdescriptions and representations are the means used by those skilled inthe data processing arts to most effectively convey the substance oftheir work to others skilled in the art. An algorithm is here, andgenerally, conceived to be a self-consistent sequence of operationsleading to a desired result. The operations are those requiring physicalmanipulations of physical quantities. Usually, though not necessarily,these quantities take the form of electrical or magnetic signals capableof being stored, transferred, combined, compared, and otherwisemanipulated. It has proven convenient at times, principally for reasonsof common usage, to refer to these signals as bits, values, elements,symbols, characters, terms, numbers, or the like.

It should be borne in mind, however, that all of these and similar termsare to be associated with the appropriate physical quantities and aremerely convenient labels applied to these quantities. Unlessspecifically stated otherwise as apparent from the discussion as setforth above, it is appreciated that throughout the description,discussions utilizing terms such as “processing” or “computing” or“calculating” or “determining” or “displaying” or the like, refer to theaction and processes of a computer system, or similar electroniccomputing device, that manipulates and transforms data represented asphysical (electronic) quantities within the computer system's registersand memories into other data similarly represented as physicalquantities within the computer system memories or registers or othersuch information storage, transmission or display devices.

The algorithms and displays presented herein are not inherently relatedto any particular computer system or other apparatus. Various generalpurpose systems may be used with programs in accordance with theteachings herein, or it may prove convenient to construct morespecialized apparatuses to perform the method operations. The structurefor a variety of these systems appears from the description above. Inaddition, the invention is not described with reference to anyparticular programming language. It will be appreciated that a varietyof programming languages may be used to implement the teachings of theinvention as described herein.

Embodiments of the invention may include various operations as set forthabove or fewer operations or more operations or operations in an orderthat is different from the order described herein. The operations may beembodied in machine-executable instructions that cause a general-purposeor special-purpose processor to perform certain operations.Alternatively, these operations may be performed by specific hardwarecomponents that contain hardwired logic for performing the operations,or by any combination of programmed computer components and customhardware components.

Throughout the foregoing description, for the purposes of explanation,numerous specific details were set forth in order to provide a thoroughunderstanding of the invention. It will be apparent, however, to oneskilled in the art that the invention may be practiced without some ofthese specific details. Accordingly, the scope and spirit of theinvention should be judged in terms of the claims that follow as well asthe legal equivalents thereof.

What is claimed is:
 1. A method for use in a portable electronic devicehaving a camera and a display screen for which the display mode isswitchable between a portrait mode and a landscape mode in response toan inertial sensor, the method comprising: capturing a first image of afeature of a user of the device with the camera; determining that thedisplay mode should be switched based on an output of the inertialsensor indicating movement of the device after capturing the firstimage; capturing a second image of the feature of the user of the devicewith the camera in response to the determining that the display modeshould be switched based on the output of the inertial sensor indicatingthe movement of the device; and disabling the switching of the displaymode determined based on the output of the inertia sensor if the secondimaged feature changes orientation with respect to the device incomparison to the first imaged feature by less than a threshold amount.2. The method of claim 1 wherein the imaged feature comprises part ofthe user's face.
 3. The method of claim 1 wherein an imaging lens of thecamera and the display screen are both disposed on a front surface ofthe device.
 4. The method of claim 1, further comprising: determiningthat a facial stabilization mode has been selected.
 5. The method ofclaim 1, further comprising: determining an initial orientation of thefeature of the user with respect to an initial orientation of thedevice.
 6. The method of claim 1, further comprising: switching thedisplay mode if the feature changes orientation with respect to thedevice by more than a threshold amount.
 7. The method of claim 1,further comprising: determining the orientation of the first imagedfeature by determining an orientation of a line passing through theuser's eyes in a plane of the first image, relative to an orientation ofthe device; and determining the orientation of the second imaged featureby determining the orientation of a line passing through the user's eyesin a plane of the second image, relative to an orientation of thedevice.
 8. The method of claim 7, further comprising: comparing thedetermined orientation of the line in the first image to the determinedorientation of the line in the second image.
 9. The method of claim 7,wherein the determining of the orientation of the second imaged featureincludes determining the orientation of a line passing through theuser's eyes in a plane of a plurality of images.
 10. The method of claim1 wherein the image is captured by the camera having an imaging lensdisposed on the same surface of the device as the display screen. 11.The method of claim 1, further comprising: using facial recognition todetect the user feature in the first and second images.
 12. The methodof claim 1, wherein the disabling of the switching of the display modeis enabled by a selection of a predetermined mode of operation.
 13. Themethod of claim 12, wherein the selection of the predetermined mode ofoperation is performed by a menu selection.
 14. The method of claim 12,wherein the selection of the predetermined mode of operation isautomatically performed by predetermined applications.
 15. A portableelectronic device comprising: an inertial sensor; a display whosedisplay mode is switchable between a portrait mode and a landscape mode;a processor for determining that the display mode should be switched inresponse to movement of the device sensed by the inertial sensor; and acamera for creating images of a feature of a user of the device; whereinthe processor is configured to control the inertial sensor, display andcamera to: capture a first image of a feature of a user of the devicewith the camera; determine that the display mode should be switchedbased on an output of the inertial sensor indicating movement of thedevice after capturing the first image; capture a second image of thefeature of the user of the device with the camera in response to thedetermining that the display mode should be switched based on the outputof the inertial sensor indicating the movement of the device; anddisable the switching of the display mode determined based on the outputof the inertia sensor if the second imaged feature changes orientationwith respect to the device in comparison to the first imaged feature byless than a threshold amount.
 16. The portable electronic device ofclaim 15 wherein the camera and display are both disposed on a frontsurface of the device.
 17. The portable electronic device of claim 15wherein the inertial sensor comprises an accelerometer.
 18. The portableelectronic device of claim 15 further comprising circuitry fortransmitting and receiving cellular network communication signals.
 19. Anon-transitory computer readable medium having program instructions,which when executed by a processor perform a method, the methodcomprising: capturing a first image of a feature of a user of a portableelectronic device using a camera of the device; determining that adisplay mode of the device should be switched between a portrait modeand a landscape mode based on an output of an inertial sensor of thedevice indicating movement of the device after capturing the firstimage; capturing a second image of the user feature with the camera inresponse to the determining that the display mode should be switchedbased on the output of the inertial sensor indicating the movement ofthe device; and disabling the switching of the display mode determinedbased on the output of the inertia sensor if the second imaged featurechanges orientation with respect to the device in comparison to thefirst imaged feature by less than a threshold amount.
 20. Thenon-transitory computer readable medium of claim 19, wherein the imagedfeature is the user's face.
 21. The non-transitory computer readablemedium of claim 19, wherein the camera and display screen are bothdisposed on a front surface of the device.
 22. The non-transitorycomputer readable medium of claim 19, wherein the method furthercomprises: determining that a facial stabilization mode has beenselected.
 23. The non-transitory computer readable medium of claim 19,wherein the method further comprises: determining an initial orientationof the feature of the user with respect to an initial orientation of thedevice.